Allow WANDB_ENTITY override via caller environment - #1

self_distillation_mask previously excluded any sample that didn't have
a successful within-group demonstration or usable feedback (gated by
reward_tensor via success_reward_threshold). But the self-distillation
loss never reads reward/advantages at all (see
compute_self_distillation_loss's argument list) — reward's only role
here was deciding which samples got distilled, conflating "the task's
reward function liked this rollout" with "this sample should be
distilled." Those are unrelated once reward is just a placeholder/proxy
rather than the actual training objective (as it is for the current
Engaging smoke test, and will be once GOOD's context replaces the
solution/feedback reprompt entirely). Every sample now participates
unconditionally.

Verified on Engaging (2x H200, Qwen3-8B, tooluse dataset): 3/3 training
steps complete, self_distillation/reprompt_sample_fraction now 1.0 at
every step (previously 1.0/0.96875/0.96875, gated by success/feedback
availability).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Replace SDPO's success-demonstration teacher with GOOD goal-context
reprompting, and validate end-to-end on 1000 WildChat conversations
(Qwen3-8B student, 16 steps, self_distillation metrics all nominal).

Teacher construction (ray_trainer.py): drop the reward-gated
_collect_solutions_by_uid / solution+feedback teacher; instead every
sample is unconditionally reprompted with GOOD's per-turn goal context,
looked up O(1) by (conversation_id, turn_index).

Goal contexts are computed OFFLINE, not live during rollouts:
- data/precompute_good_contexts.py walks each conversation's turns in
  order (GOOD state is incremental) and writes a flat
  {conversation_id:turn_index -> context} JSON. Resumable via atomic
  temp-file writes; --provider {openrouter,vllm}.
- data/vllm_provider.py implements GOOD's LLMProvider Protocol against a
  self-hosted vLLM (Qwen3-32B chat + Qwen3-Embedding-8B), so GOOD's
  shared-prefix comparison batches hit vLLM prefix caching (~84%) at
  GPU-hours we own instead of per-token OpenRouter.
- verl/utils/good_state_cache.py serves the table from a read-only
  detached Ray actor: no OpenRouter / good_goals / API key at train time.

Data (wildchat_chop_dataset.py): one logical row per conversation,
resampling the chop turn each __getitem__ so epochs re-draw chops instead
of letting long conversations dominate a static explode. Candidate turns
are filtered to those fitting max_prompt_length (the agent-loop rollout
path does not cap length, and the parent's static filter is meaningless
for a dynamic chop).

actor.yaml / actor.py: add self_distillation.good_contexts_path.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

SDPO's teacher is a separate full copy of the model (ref_module_fsdp, EMA-updated
toward the student). _is_ref is role-based rather than LoRA-gated, so a LoRA run
would still allocate that second copy -- silently paying the exact memory cost
LoRA exists to avoid, while appearing to work.

Add teacher_regularization="frozen-base": the teacher becomes the student's own
base with adapters disabled (LoRADisabledTeacher), so no ref model is built and
the teacher costs nothing. Semantically this is context distillation with a fixed
teacher; verl already uses the adapter-disabled-base trick for the reference
policy in compute_log_prob. The worker now raises rather than letting a LoRA run
quietly build the redundant copy.

Also adds data/split_conversations.py: training so far pointed train_files and
val_files at the same conversations.json, leaving no held-out data. Splits on
whole conversations, since WildChatChopDataset redraws the chop point per epoch
and a turn-level split would leak.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…context

vllm_async_server unconditionally overwrote config.max_model_len with the
model's max_position_embeddings, making the rollout.max_model_len setting
silently inert -- the override showed up in the Hydra override list while
vLLM still reported max seq len 40960.

This deadlocks large models. vLLM reserves KV cache for one request at
max_model_len and refuses to start if the memory left after weights cannot
cover it; lowering gpu_memory_utilization to buy FSDP headroom shrinks that
same pool, so the two knobs fight with no setting that satisfies both. The
32B LoRA run could not start at 2x H200: 5.00 GiB KV needed vs 4.17 GiB
available (jobs 19728636, 19736173, 19767821, 19768117).

Now the model default applies only when max_model_len is unset, and an
over-large value is clamped with a warning.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Wires GOODConfig.comparison_leader_weight (good-goals b5c8401) through
precompute_good_contexts.py as --comparison_leader_weight (default 0.0 = prior
behavior). The afterfix production regens run with 2.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…st quality eval

Additive: training path (build_teacher_messages) untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…prompt util, WC datasets

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>